Quantile regression (Koenker and Bassett, 1978) is a popular tool in statistics and econometrics. The method extends median regression from fitting the conditional median to modeling a suite of conditional quantile functions, providing a more comprehensive and nuanced view of the relationship between a response variable and its predictors. One of the key advantages of quantile regression, also rooted in median regression, is its robustness against outliers in the response direction. Since its introduction, quantile regression has been adapted in various research areas, including survival analysis (Peng and Huang, 2008; Wang and Wang, 2009), longitudinal data modeling (Koenker, 2004), machine learning (Meinshausen and Ridgeway, 2006; Fakoor et al., 2023), and so on, and has seen widespread applications in fields such as finance, ecology, healthcare, and engineering. For detailed introductions and the latest developments in quantile regression, see Koenker (2017) and Koenker et al. (2018). Despite its popularity, one primary limitation of quantile regression is its high computational cost, which is also inherited from its median regression origins.

Language models (LMs) have proven to be powerful tools for psycholinguistic research, but most prior work has focused on purely behavioural measures (e.g., surprisal comparisons). At the same time, research in model interpretability has begun to illuminate the abstract causal mechanisms shaping LM behavior. To help bring these strands of research closer together, we introduce CausalGym. We adapt and expand the SyntaxGym suite of tasks to benchmark the ability of interpretability methods to causally affect model behaviour. To illustrate how CausalGym can be used, we study the pythia models (14M--6.9B) and assess the causal efficacy of a wide range of interpretability methods, including linear probing and distributed alignment search (DAS). We find that DAS outperforms the other methods, and so we use it to study the learning trajectory of two difficult linguistic phenomena in pythia-1b: negative polarity item licensing and filler--gap dependencies. Our analysis shows that the mechanism implementing both of these tasks is learned in discrete stages, not gradually.

Interactive Fiction games are text-based simulations in which an agent interacts with the world purely through natural language. They are ideal environments for studying how to extend reinforcement learning agents to meet the challenges of natural language understanding, partial observability, and action generation in combinatorially-large text-based action spaces. We present KG-A2C, an agent that builds a dynamic knowledge graph while exploring and generates actions using a template-based action space. We contend that the dual uses of the knowledge graph to reason about game state and to constrain natural language generation are the keys to scalable exploration of combinatorially large natural language actions. Results across a wide variety of IF games show that KG-A2C outperforms current IF agents despite the exponential increase in action space size.